The present invention relates to computer systems, and more particularly, to techniques for detecting and preventing malicious code execution within a computer system.
There is no way to completely categorize all forms of attacks on software systems, but the most common attack techniques can be considered as either (1) application-level attacks employed by Internet crackers and worms, and (2) denial-of-service attacks and attacks that exploit ‘logic errors” in the software to cause race conditions or otherwise change the execution of the program.
The first category, application level attacks, exploit vulnerabilities in the software, and force the system to run code of the attackers' choice. This form of attack is the focus of the embodiments described herein. The attacks within the second category require different defense methods, which are outside the scope of the embodiments described herein.
Launching a successful attack that exploits security flaws in software with the intent of compromising the running software has typically involved (a) the attacker overriding the flow of control in the program's execution, and redirecting control to malicious code, i.e., code of the attacker's choice, and (b) introducing malicious code into the program memory.
Control Overriding
Application level attacks generally involve buffer overruns or format-string exploits to overwrite crucial locations in the program memory that contain “code pointers.” These code pointers include the return-address and the old-frame-pointer values stored in activation records on the program stack, function-pointer variables, and entries in the procedure-linkage (PLT), global-offset (GOT) and virtual function-pointer (VPTR) tables in the heap, .bss (un-initialized static area) and the initialized static data areas. What is special about these code pointers is that the control flow of the program is directly affected by the values stored in these locations, e.g., the return-address value determines what code is executed after a function has completed execution and needs to return control to its caller. Attackers exploiting buffer-overrun and format-string vulnerabilities try to write the address of “malicious code” into these memory locations, with the aim of getting the processor to start fetching and executing instructions from the malicious code at some point.
The chief factor that enables a malicious attacker to exploit software flaws to force a control override in the program execution is the low-level, data-handling power afforded by systems languages like C and C++. Specifically, the lack of bounds checking on pointer arithmetic (array indexing is implemented as pointer operations) for these languages permits write operations to virtually any part of data memory. Consequently, poor programming practices can lead to a failure on the part of application developers to explicitly validate the size and/or format of the input data. This results in buffer overflows, where input data is copied past the end of the memory space reserved for the data copy, and onto adjoining data. Unfortunately, it is not just application code that is likely to suffer from such vulnerabilities—core routines in the C standard library (libc) such as printf, gets, syslog, and strcpy cannot make assumptions about the size of input and/or output data, and hence are implemented to blindly copy data from the memory location depicted by the input parameter until this data is exhausted. Such buffer overflows can cause severe memory corruption and possibly distort the execution of the program. It is not surprising that buffer overflows are perhaps the most widely exploited type of vulnerability, accounting for approximately half the CERT/CC Advisories in recent years (see, www.cert.org/advisories/).
Malicious Code
A large class of successful attacks on software systems involves some form of injection of executable code via data I/O channels. Once the malicious code is resident in the program memory, all the attacker needs to do is to trigger the injected code via one of the mechanisms for overriding the control flow of the program, described in the last section. However, given the advent of defense techniques that are capable of detecting code executing in memory normally reserved for data, attackers have sought to redirect program control not just to injected code, but rather to existing routines such as system and execve in the standard library. Such “return-into-libc” attacks avoid the need to both inject executable content into data areas of the program memory and override the program flow of control. There is, although, more work involved with these attacks since they need to “massage” the input parameters to these routines in order to make the effect of invoking these routines truly malicious.
It should be noted that injected code and certain functions in the standard library are not the only forms of code that can be abused by return-into-libc-style attacks—any program function can serve the attacker's purposes if (a) he has the liberty of specifying its input parameters, and (b) it ultimately invokes some critical library functions or system-call routines. The following illustrates a wrapper function that introduces vulnerabilities in this manner:
1. void execute(char *program) {2. /· this is my own wrapper to system */3. system(program);4. }
Such programmer-created vulnerabilities may not necessarily be the result of bad programming practices, but could possibly be intentionally inserted by a disgruntled employee or other malicious insider who has access to the codebase.
The following techniques make up the majority of attacks on software systems today.
Stacksmashing Attack—In a demonstration of the classic “stacksmashing” attack technique, Aleph One [“Smashing the stack for fun and profit,” Phrack, 7(49), 1996] provides a detailed walkthrough of how to exploit a stack buffer-overflow vulnerability to inject attack code, and also overwrite the return address of the function to point to the starting address of this injected code. This paper also suggests ways to increase the chances of the stacksmashing attack:                pad the beginning of the injected code with no-op instructions to make the exploit work even when the exact address of the injected code is not known, and        approximate the actual position of the return address relative to that of the vulnerable buffer by copying the address of the injected code over a range of locations, hopefully including the return address position        
The following illustrates a modified version of the original stack-smashing attack described by Aleph One, where malicious code is injected onto a stack buffer, and the buffer is overrun to overwrite the return address of the function, causing an override of the control flow of the program.
1  char *shellcode =2   “\xeb\xif\x5e\x89\x76\x08\x31\xc0”3   “\x88\x46\x07\x89\x46\x08\x0c\xb0”4   “\x89\xf3\x8d\x4e\x08\x8d\x56\x0c”5   “\xcd\x80\x31\xdb\x89\xd8\x40\xcd”6   “\x80\xe8\xdc\xff\xff\xff\bin\sh”78  int main(void) {9   char buffer[96];1011   int I=0, *p = (int *) buffer;12   while (i++ < 32) *p++ = (int) buffer;13   strncpy(buffer, shellcode,14    strlen(shellcode));15   return 0;16  }
When the function completes execution and attempts to return control to its caller, the processor will fetch and execute instructions from whatever return address is specified in the corrupted activation frame header. In a successful attack, this value will typically point to the base of the injected code, i.e., the address of the stack buffer. The while-loop in line 12 above writes 32 (4-byte) words into the 96-byte memory block (buffer) pointed by the pointer (p), thus overflowing buffer and causing the beginning address of the attack code to overwrite the return address for the function. The strncpy statement in lines 13 and 14 completes phase two of the stack-smashing technique, injecting the attack code into the buffer. This injected “shell-code” represents a sequence of Pentium x86 machine instructions to invoke a system call to spawn a shell.
A slight variation on this theme involves injection of code into non-stack buffers, e.g., buffers in the heap, .bss (un-initialized static data) or static initialized data sections. In the case of these static data areas, it is easier for the attacker to determine the exact location of the target buffer, and hence the address of the injected code, since the absolute addresses are determined at link time. Yet another variation involves overwriting, at least part of the stored frame-pointer value in the activation record—this forces the attacked function's caller function to trigger the injected code.
Function-Pointer Overwriting
This form of attack is similar to the stack-smashing technique in that it involves the use of an unsafe operation with unchecked input data. The difference lies in the fact that the buffer overflow overwrites some other data that is capable of altering the control-flow of the process rather than overwriting the return address or the stored frame-pointer value in an activation record. Various forms of such attacks involve overwriting function pointer variables in C structures, in function activation records, or in C++ classes. It is also possible to overwrite entries in the virtual function-pointer table used for implementing polymorphism in C++, or in the global offset table (GOT), to redirect control to malicious injected code.
In these scenarios, it is not the return from a function body that triggers the attack code, but rather an explicit function invocation (via a function pointer or virtual function) that triggers the injected code. These attacks are harder to prevent than the stack-smashing technique, and are often able to circumvent the various stack-based defense mechanisms known in the art.
Format-String Exploit
this Attack Technique is Unique in that it does not involve an explicit memory overwrite via any of the unsafe copy operations listed above. Instead, it exploits the ‘/.n modifier in C's format strings capability, when combined with dynamically generated strings, to overwrite specified locations in memory with arbitrary values. Obviously, this form of attack can be used to circumvent the normal control-flow by overwriting any of the stored return-address in the activation record, function-pointer variables or entries in various function-pointer tables.
Return-into-Libc Attack
Many attack techniques involve some sort of code-injection to alter the normal control-flow of the application. There are, however, other forms of attacks, known as “return-to-libc” attacks, that leverage existing functionality to cause malice to the system. This form of attack also involves overwriting of code pointers. However, instead of pointing to any injected attack code, these pointers are overwritten to point to the address of libc functions, typically wrapper functions that provide an indirection to system-call functions. This attack mechanism is able to reuse existing code in the attacked system to bypass most injected-code detection mechanisms, and is undoubtedly much more difficult to beat.